Encode content before calling LWP #27

ksmadsen · 2014-11-04T20:58:34Z

Hi,

This issue caught us today. I hope that this patch with a test-case is deemed appropriate.

When LWP encodes a HASH-ref, as is done in generic_solr_request it will use the
URI module to create the www-form-urlencoded content from the HASH-ref.

The URI module will try to deduce the desired target charset from the utf8-flag
on the strings, which means that the strings sent to Solr aren't UTF-8 encoded
unless they have been either decoded, or encoded as utf8.

This is confusing, as e.g. the string "\xc6" is passed on to Solr as an UTF-8
encoded Æ passed through the add-method, but the same string will result in an
error from Solr if used with the search-method.

To fix this issue, encode all of the strings from the parameters in
generic_solr_request before passing them on to LWP. This way the charset
behaviour of generic_solr_request and _send_update is aligned.

Note: This will break applications that encode strings to UTF-8 before calling
WebService::Solr generic_solr_request, search or auto_suggest.

When LWP encodes a HASH-ref, as is done in generic_solr_request it will use the URI module to create the www-form-urlencoded content from the HASH-ref. The URI module will try to deduce the desired target charset from the utf8-flag on the strings, which means that the strings sent to Solr aren't UTF-8 encoded unless they have been either decoded, or encoded as utf8. This is confusing, as e.g. the string "\xc6" is passed on to Solr as an UTF-8 encoded Æ passed through the add-method, but the same string will result in an error from Solr if used with the search-method. To fix this issue, encode all of the strings from the parameters in generic_solr_request before passing them on to LWP. This way the charset behaviour of generic_solr_request and _send_update is aligned. Note: This will break applications that encode strings to UTF-8 before calling WebService::Solr generic_solr_request, search or auto_suggest.

petdance · 2014-11-05T00:12:17Z

I can't even begin to assess if this is something that we should put in. Encodings are far from my strong suit.

tonycoz · 2019-06-14T05:35:36Z

something like this is needed, for example:

#!perl
use strict;
use warnings;
use WebService::Solr;

binmode STDOUT, ":utf8";

my $text = "alpha\xFF";

# default managed schema includes id and _text_
my $doc =
  {
    id => "abc",
    _text_ => $text,
  };

my $solr = WebService::Solr->new("http://127.0.0.1:8983/solr/test");
$solr->ping
  or die;

$solr->add($doc)
  or die;

use Data::Dumper;
my $resp = $solr->search($text);
$resp->is_success
  or die Dumper($resp->content);

results in:

$VAR1 = {
          'error' => {
                       'metadata' => [
                                       'error-class',
                                       'org.apache.solr.common.SolrException',
                                       'root-error-class',
                                       'org.apache.solr.common.SolrException'
                                     ],
                       'msg' => 'URLDecoder: Invalid character encoding detected after position 10 of query string / form data (while parsing as UTF-8)',
                       'code' => 400
                     }
        };

Note that the text is being encoded correctly during indexing because _send_update() encodes the document before sending:

    HTTP::Headers->new( Content_Type => 'text/xml; charset=utf-8' ),
    '<?xml version="1.0" encoding="UTF-8"?>' . encode( 'utf8', "$xml" )

while search() doesn't.

karenetheridge · 2020-01-06T22:39:30Z

related - libwww-perl/URI#48

petdance added the Bug label Dec 10, 2016

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Encode content before calling LWP #27

Encode content before calling LWP #27

ksmadsen commented Nov 4, 2014

petdance commented Nov 5, 2014

tonycoz commented Jun 14, 2019

karenetheridge commented Jan 6, 2020

Encode content before calling LWP #27

Are you sure you want to change the base?

Encode content before calling LWP #27

Conversation

ksmadsen commented Nov 4, 2014

petdance commented Nov 5, 2014

tonycoz commented Jun 14, 2019

karenetheridge commented Jan 6, 2020